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Programed Instruction 


Read all about it! See it now! 
Learn by doing! Let’s stay experimen- 
tal! These were recurring themes in 
programing as the academic year got 
into full swing. Highly informative 
articles on the field appeared in the fall 
issues of a number of popular profes- 
sional periodicals. The National Edu- 
cation Association included a special 
journal feature on teaching machines 
and programed learning in its No- 
vember issue. In the November Sci- 
entific American a wonderfully il- 
lustrated article by B. F. Skinner re- 
lated the language of the laboratory to 
the art of pedagogy; at the same time 
readers of the November 18th issue of 
the Saturday Review of Literature 
were being taken “Inside the Teach- 
ing Machine” by Susan Meyer Mar- 
kle. For those who were concerned with 
the subtle issues involved in “Evalu- 
ating Teaching Machines and Pro- 
grams,” James Holland’s article in the 
October issue of the Teachers College 
Record provided much _ food for 
thought. 

The American Management Asso- 
ciation, Educational Records Bureau, 
and Purdue University conferences on 
programed instruction and_ teaching 
machines provided a wide range of 
speakers to hear, and models of ma- 
chines to see. One of the most inter- 
esting was a device, using microfilmed 
programs, to retail for $80 to $90. A 
major commercial concern which has 
been selling programs door to door 
introduced its new division for mar- 
keting programs to schools. A real 
treat for the Purdue Conference go- 
ers was the opportunity to hear S. L. 
Pressey, “grandfather” of the teaching 
machine movement. 

School systems, large and small, ur- 
ban and rural, provided one-day work- 
shops and speakers to introduce their 
faculty, parents, and school board 
members to programed instruction. 

A Saturday morning in-service work- 
shop for 80 faculty members was re- 
cently completed by the Bethlehem 
Central School System in Delmar, 


The Center 


Ring Out, Wild Bells. .... 


New York. Five two-hour sessions 
were held under the tutelage of Dr. 
John Blyth of Hamilton College. 

“Workshop Revisited’ will be the 
theme when Oregon College of Edu- 
cation plays host for two days in Janu- 
ary to the participants in its summer 
workshop in programed instruction. 
Dr. Evan Keislar and Dr. Harry Sil- 
berman will likely be at the helm again. 

The Center, to date, has received 
31 requests from colleges to conduct 
workshops similar to those offered last 
summer. Soon to be announced—the 
Center’s summer institutes to train 
faculty who in turn will conduct work- 
shops and in-service courses in their 
respective academic communities. 

A steady flow of reports on research 
and use of programed materials con- 
tinues to be received at the Center. 
The Baltimore County Public Schools 
reported on the first year of a three- 
year study carried on in cooperation 
with the University of Virginia and 
involving 340 elementary school chil- 
dren. Fourth graders in six elemen- 
tary schools worked through an arith- 
metic program designed in a linear, 
constructed-response, ruleg mode, and 
written by the school system’s teach- 
ers. A second exposure to the mater- 
ials was provided. “The experimental 
treatment did not in itself handicap 
the progress of individuals or groups 
of pupils in arithmetic.” This year the 
same students will work on a program 
in 5th grade arithmetic. 

The Baldwin-Whitehall School Dis- 
trict, located outside of Pittsburgh, has 
started on a five-year study exploring 
implementation of programed materials 
at the junior high school level. Dr. 
Robert Glaser of the University of 
Pittsburgh’s Psychology Department is 
directing the project for the U.S. Of- 
fice of Education. Approximately five 
hundred 7th and 8th graders in two 
junior high schools are “testing and 
digesting” generally accepted 9th 
grade general science materials. Ad- 
ministrators quote as “highly favor- 
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Inside 
Opinion 


QUESTION: How does a program differ from a 
standard workbook? 


ANSWER: A program is a Self-contained se- 
quence of instructional items which can be 
presented to individual students under con- 
trolled conditions. It is designed specifically to 
teach material and is repeatedly tested and 
revised until it gives consistent and accepta- 
ble results. 

The items in a standard workbook provide 
practice on material taught elsewhere. A 
workbook is not designed to do a teaching job. 
Applying some programing techniques, how- 
ever, might result in making a workbook a 
good deal more efficient. An item analysis of 
students’ responses would probably lead the 
author to rewrite a number of unnecessarily 
confusing items. It might also show up unsus- 
pected areas of ignorance for which new prac- 
tice exercises could be written. The attempt to 
give students more adequate preparation for 
the components of each new exercise might 
suggest changes in the order of the exercises. 

If this process of revision were repeated, 
under adequately controlled conditions, the 
workbook might eventually approximate a pro- 
gram of instruction. The contrast between such 
a “programed workbook” and the more usual 
‘programed textbook” might help to break us 
free from the notion that all programs must 
consist of sequences of sentences with words 
left out. 

Wells Hively 


Research Fellow, Teaching Machine Project, Harvard 
University Psychological Laboratories. Author of articles 
pertaining to the field of teaching machines and pro 
gramed instruction. 


ANSWER: Aside from the obvious differences 
—- the meticulous analysis of behavior prior 
to preparation of a program, the application of 
behavioral principles of stimulus control, and 
so on, there is the overriding difference con- 
cerning the assumption of the responsibility 
for whether or not the student learns. Al- 
though | must quickly disqualify myself as an 
expert in conventional workbooks, it seems 
clear that workbook writers assume about the 
same amount of responsibility for the stu- 
dent’s behavior as textbook writers do. All 
of us reading this Bulletin have learned from 
conventional workbooks and from conventional 
textbooks. If the appropriate behavior is in the 
student’s repertoire, everything is fine. If it 
is not, nothing happens, except that extinction 
occurs. 

That the two approaches differ in the order 
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Quotable Quotes 


“Programed learning offers consider- 
able promise as a means of breaking 
a serious bottleneck in introducing 
new subject matter into the curricu- 
lum. Let me refer, for example, to 
the current revolution in mathematics. 
The bottleneck in this instance is the 
re-education of teachers. Under pres- 
ent conditions, the new mathematics 
cannot possibly be introduced into the 
curriculum of a particular school until 
the teachers understand the changes in 
content and methodology which are in- 
volved. Most of the new systems of 
instruction such as the U.I.C.S.M. 
and §.M.S.G., recognize the necessity 
of teacher education and will not 
normally make their materials avail- 
able to schools unless the teachers in- 
volved undergo a period of orientation. 
What with the ever present problem 
of teacher turnover, this makes the in- 
troduction of modern mathematics tor- 
tuously slow. If much of modern math- 
ematics can be programed, and there al- 
ready is evidence that it can be, then 
one of the major bottlenecks may be 
broken. 

It seems to be increasingly evident, 
therefore, that the major responsibility 
for the future of programed learning 
rests primarily with public school edu- 
cators like myself. We cannot afford 
to wait for the academic schools, the 
professors of education or our profes- 
sional associates to tell us what to do. 
We must take the initiative, for if we 
don’t it will be seized by others less 
qualified than we are to decide what 
should happen in the schools.” 


J. BERNARD EVERETT 


. 
Assistant Superintendent of Schools, Division of Instruc- 
tion, Newton, Mass., from a paper delivered at the 
Frontiers of Science Conference, Oklahoma City, Febru- 
ary, 1961. 
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INSIDE OPINION 














esponses 





(This month’s responses were received 
from the Director of the project using pro- 
gramed materials in Roanoke City Public 
Schools, Roanoke, Virginia last spring. 
When asked “Can and should teachers write 
programs?” the Director in turn asked a 
number of teachers involved in field trials 
of the materials. These teachers replies are 
presented not as an endorsement for a par- 
ticular method of developing programed 
materials, but as the viewpoints of one 
group of teachers, who used materials de- 
veloped by one method.) 


We agree that the subject matter 
specialists should continue to do the 
basic work in preparing the programs. 
Surely no teacher should be expected 
to prepare his own program. However, 
we feel that teachers who have actu- 
ally had classroom teaching in a par- 
ticular subject should review the ma- 
terial before it is put in the classroom. 

Margaret Fisher 
Berta Hartman 
Florence Stump 


In my opinion subject matter spe- 
cialists should prepare the programed 
material for high school courses. It is 
a task far too time consuming for any 
high school teacher. A broad knowl- 
edge of the subject matter together 
with the psychological know-how of 
presenting it is necessary. Obviously, 
then, good programs must be produced 
by specialists in the field. “Teachers 
may be able to give helpful sugges- 
tions for revision of the materials they 


have used or are using. 
Mabel Noell 


One using programed materials wii! 
develop the ability to perform mathe- 
matical operations skillfully and with 
understanding. The learner is also pro- 
vided with a rich variety of exper- 
iences to assure individual success. If 
these objectives are to be achieved, the 
materials used should be the findings 
of most recent research. This I believe 
can be done effectively by specialists. 
Because the teacher has daily contacts 
with the use of this material, her role 
in the total evaluation cannot and must 
not be minimized. Her critical analyses 
will be invaluable to the specialist, in 
his preparation, in meeting the needs of 
the individual for further programing. 


Dorothy D. Witten 
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(Continued from page 1) 


of teaching power is, | believe, attested to by 
the fact that there is now in existence, all 
observers agree, a ‘teaching machine move- 
ment.” Again, although | am not a workbook 
expert, | can recall no ‘workbook movement” 
being generateu. 

But the only really definitive test of the 
difference between a workbook and a program 
is, in the final analysis, the differences in the 
effects they have on student behavior. 


Lloyd E. Homme 
President, Teaching Machines Incorporated, Albuquerque, 


New Mexico. Pioneer author, researcher of programed 
instructional materials and teaching machines. 








Teaching Plane Geometry by the 
use of the programed materials as pre- 
pared by the specialists in the field has 
been gratifying. The course is desira- 
bly planned; it is very inclusive in the 
choice of subject matter—intensive and 
extensive. Ihe average classroom teach- 
er would not have the time to perform 
the task as effectively as it has been 
done and neither could she make the 
improvements as rapidly as they are 
being incorporated. The teacher needs 
her time for other teaching duties that 
are uniquely hers. It is hoped that the 
programed materials will continue to 
be prepared by the specialists just as 
textbooks are now being written. 


Ella M. Keen 


Previous to the use of the experi- 
mental mathematics this year, I had 
been convinced that experienced teach- 
ers were the ones best informed and 
most able to prepare programed ma- 
terials. After having used several of 
the mathematics programed textbooks, 
I have changed my opinion and am 
thoroughly convinced that the special- 
ist in the subject matter field is super- 
ior and that the teacher’s place is of 
considerable importance in the revi- 
sion. The teacher can make construc- 
tive suggestions and I believe that the 
specialist and teacher together make up 
the best possible combination. 

Martha Walden 


A subject matter specialist working 
closely with naive subjects seems to be 
the best way to produce good pro- 
grams. Such a specialist will be able to 
reword material that proves to be 
dificult for the student—or even ap- 
proach the material from a different 
angle without losing sight of the ob- 


jective. 
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Two types of experiments in programing seem to have 
the greatest appeal for educators: (a) the comparison of 
machine and textbook presentation of programed mater- 
ials and (b) the comparison of programed and conven- 
tional instruction. Neither type, unfortunately, is prac- 
tical or would be useful at this stage. 


The machine versus textbook experiment is expensive, 
because machines have to be purchased or rented for the 
purpose; and programs usually available for machine pre- 
sentation are not necessarily available in text format. 
More important than financial considerations is the fact 
that seven previous studies have shown no significant 
differences in learning between machine and textbook 
presentations of programs. In most of the studies there 
was a Significant saving in time in favor of the pro- 
gramed text. These results should not be interpreted to 
mean that there is no purpose in investigating the efficacy 
of machines, but rather that research in this area should 


be refined. 


Educators also wish to pit teachers against programs 
—to compare programed instruction with the “conven- 
tional method” of teaching. Comparisons between ‘‘con- 
ventional methods” and programed instruction cannot be 
generalized because they must reflect the competence of 
the particular teacher, the quality of the program used, 
and the evaluative instruments. In some cases the teacher 
who is pitted against the program is stimulated to su- 
preme effort by the apparent threat of being replaced by 
a teaching machine. It is not surprising that contradictory 
findings evolve from such research. Programs have been 
found to out-teach teachers and vice versa. Even if re- 
sults were clear-cut, nothing would be proved, because 
programed instruction is designed to be used by teachers. 

Neither type of study can be recommended as an ef- 
fective way for educators to learn what programed in- 
struction is or to determine its place in the educational 
enterprise. The experimentation described here deals 
with ways in which interested educators can learn about 
unique attributes of programed materials, it does not 
represent CPI’s views of basic research needed. 

More reasonable approaches to experimenting with 
programed instruction include (a) Case Study, (b) Field 
Testing, (c) Implementation. This article explores the 
first two approaches. 


Case Study 

In the case study approach the educator examines the 
application of programed instruction for an individual 
student with a particular problem or in a particular situ- 
ation. The most obvious application is for gifted and ex- 
ceptional children. A 5th grade teacher may have an 
extremely bright student who already grasps 5th grade 
arithmetic. He might put the student on an advanced 
programed unit, like Richard Smith’s Prime Numbers 
and Factoring, developed in the Newton, Massachusetts, 
school system. The teacher can then report on this stu- 


dent’s reaction to working with such materials and ana- 
lyze the student’s performance on a posttest. 


A high school teacher might be interested in trying 
Susan Markle’s vocabulary units with a high school jun- 
ior who has difficulty with words. The question to be 
answered is how a student, two to four years below grade 
level in vocabulary and reading will react to programed 
materials designed and tested for 8th graders. 


A school or school system exploring the potential of 
programed instruction might have several interested 
teachers choose a program, then try it with a selected 
student, and compare their findings and observations. 
The success of the case study approach obviously depends 
upon the program selected. If it becomes apparent that 
the program fails to stimulate or is too difficult, teachers 
should not hesitate to try another program. 

This approach simulates the method used by the labor- 
atory psychologist seeking to chart the learning of a 
single organism. Limiting initial experimentation to one 
learner provides opportunity for careful attention to the 
entire process—from assessing the student’s behavior dur- 
ing acquisition (examining his responses and attention 
span while working on the program), to measuring the 
terminal behavior. 


Field Testing 


As Ernest Rothkopf of Bell Laboratories stated at the 
November meetings of the Educational Records Bureau, 
programs should come with a “label” indicating how 
they have been tested and the results obtained. Even if 
the program selected has been tested with a group of 
students, much can be learned from repeating the test 
with part of the program in a few classroom periods. 
Such field testing asks: Does the program teach the con- 
tent it purports to? Is the content well programed? 
Which students learn from it? How much time do stu- 
dents need to complete the program? 

To assess what the program teaches, students must be 
tested after completing the program. Often, a programer- 
constructed test will be supplied with the program. The 
consumer should analyze the program to assess the ade- 
quacy of the criterion test. Careful analysis of errors, by 
items and by individuals, on both the program and the 
test, provides important insights into programing quality 
and individual differences. 

The responses to a sequence from an entire class will 
reveal administrative problems: How to provide for the 
student who finishes in one third of the time that the 
slower student takes? How to deal with that portion of 
the class that cannot handle the program? What to do 
about those students who are so high on the pretest that 
they would not profit from the program? (For purposes 
of exploring the field these students should work the 
program anyway.) Does the particular unit seem more 
appropriate for the more able or less able students? Such 


Continued on page 





RESEARCH 


Kalin, K. Development and Evaluation 
of a Programed Text in an Advanced 
Mathematical Topic for Intellectually 
Superior Fifth and Sixth Grade Pu- 
pils. (Doctoral dissertation abstract) 
Florida State University, 1961. 


This study investigated the ability 
of “intellectually superior” pupils to 
learn advanced mathematics through 
programed instruction. 


Subject Matter: 
Mathematics. “Equations and In- 
equalities.” Programed text. 


Subjects: 

Ninety-five 5th- and 6th-grade stu- 
dents, all above average (IQ greater 
than 115, reading grade-placement 
greater than 5.0, and arithmetic grade- 
placement greater than 5.5). The pu- 
pils were randomly assigned to ten 
groups, five experimental and five con- 
trol. 


Procedure: 

Experimental subjects read the pro- 
gramed text at own rate, without as- 
sistance. Control subjects were taught, 
with conventional methods, by teach- 
ers who were “prepared” by confer- 
ences, instruction, and study of the 
program. A 44-item criterion test was 
the evaluation instrument. 


Results: 

Differences in test score between 
paired experimental and control, were 
not statistically significant. “Twenty 
per cent time saving for experimental 
groups. Means for these 5th and 6th 
graders rangers from 26.2 to 30.8 com- 
pared with a 9th grade algebra class 
mean of 35.0. 


Comment: 

The author asserts that in- 
tellectually superior fifth- and sixth- 
grade pupils can learn advanced math- 
ematics from a _ programed text.” 
Granted. But “intellectually superior” 
individuals can learn almost anything. 
Perhaps it would be more interesting 
to determine whether programed in- 
struction can teach less gifted persons 
subject matter considered to be beyond 
their ken, 


ce 





























Evans, J. L., Glaser, R., and Homme, 
L. E., “An Investigation of ‘Teaching 
Machine’ Variables Using Learning 
Programs in Symbolic Logic. Depart- 
ment of Psychology, University of 
Pittsburgh, Pennsylvania. December, 
1960. 


This study investigated (a) the ef- 
fects of differences in response modes 
(constructed, multiple choice, and no 
overt response) ; (b) the value of im- 
mediate reinforcement or confirmation 
of responses; (c) the value of a for- 
mal rule for the construction of pro- 
grams, 


Program: 

Symbolic Logic. Two programs 
were constructed to teach 15 rules 
from the calculus of propositions. The 
Initial Program of 72 frames was con- 
structed based on “standard” pro- 
graming procedure. The Formal Pro- 
gram of 125 frames used the “ruleg” 
system of cue-withdrawal, in which 
the rule is first described with an ex- 
ample to be completed, then stated 
incompletely with the subject required 
to complete it, then offered in the 
form of an incomplete example to 
which it must be applied. The re- 
quired terminal behavior was the use 
of the 15 rules to construct short de- 
ductive proofs. 


Subjects: 
60 University of Pittsburgh  stu- 
dents paid $+ each for the experiment. 


Procedures: 
The students were randomly divid- 
ed into six treatment groups. 


Four groups used the Initial Pro- 
gram: (1) Constructed Response; (2) 
Multiple-Choice Response; (3) No 
Overt Response; (4+) Reinforcement 
within the frame. 


Two treatment groups used the 
Formal “ruleg’” Program: (1) Con- 
structed Response; (2) Constructed 
Response with a cue constantly present 
(a panel listing examples of the fifteen 
rules. ). 


Three tests were given: (1) True- 
False; (2) Performance in use of 
rules; (3) Performance in construc- 


Ce ee 


tion of deductive proofs by combina- 
tions of rules. Time to complete the 
program and time to complete tests and 
delayed posttest were recorded. 


Results: 

1) Students not required to make 
an overt response required less time to 
complete the program and scored as 
well on the criterion test. 


2) Students in the “ruleg” formal 
program required significantly _ less 
time (though there were more frames 
in the program), and achieved rough- 
ly the same test scores as students on 
the less systematic program. 

3) Retention was influenced by the 
kind of test administered. Scores on 
the true-false test improved after a 
week; scores on recall tests showed a 
small but significant drop. There was 
no significant change in the scores in- 
volving deductive 
proofs. 


construction of 


4) Test scores did not correlate 
with previous mathematical experience, 
college class, nor with degree of re- 
tention. 


Comment: 

Use of Symbolic Logic as subject 
matter seems highly appropriate to the 
research. The construction of deductive 
proofs provides .an unusually good 
measurement of transfer. The finding 
that retention varied as a function of 
the kind of test is extremely important 
to the search for criterion measures. 


The comparison between the “‘nor- 
mally” and formally constructed pro- 
grams is interesting, but the results 
may be interpreted as displaying a 
weakness in the normally constructed 
program employed, or as a tribute to 
the unique suitability of Symbolic Log- 
ic for the “ruleg’’ approach, rather 
than as proof of the superiority of a 
formally constructed program. 

With these considerations in mind, 
it should be pointed out that the au- 
thors have given the first existence 
proof that one program can be con- 
structed by means of a formal system 
which will teach as effectively as a 
“programer  constructed’’ program. 
This is an important first step in the 
development of programing as a sci- 
ence as opposed to the “technological 
art” that it is today. 

(Continued on following page) 
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Alter, M., Eigen, L., and King, S. “A 
Preverbal Number Concept Program.” 
Center for Programed Instruction, New 
York, New York. 1961. 


These studies aimed to measure the 
efficiency of a preverbal program, un- 
der “confirmation” and ‘confirmation 
and trinkets” reinforcement conditions, 
with preliterate young children. 


Study I 
Program: 
Number concept. 262 frames; mul- 
tiple-choice response. 


Subjects: 

Sixteen children, 5 and 6 years of 
age, students at the Dalton School 
(New York City). 


Procedures: 

Each frame, consisting of a top or 
“stimulus” portion and a lower por- 
tion containing three alternate “‘re- 
sponses,” was administered to each 
subject individually. Subjects were re- 
quired to make three types of matches: 
(a) number of objects to number of 
objects, (b) digit to number of ob- 
jects, and (c) number of objects to 
digit. Similarity of objects and color 
of objects were used as cues. On the 
basis of a pretest (21 frames in the 
same format as the program), subjects 
were either eliminated or assigned to 
one of two experimental groups. Re- 
sponses of “confirmation” subjects were 
immediately confirmed or corrected 
before they proceeded to the next 
frame. ‘Confirmation and_ trinket” 
subjects, in addition to this procedure, 
were given tiny prizes for approxi- 
mately 25 per cent of their correct re- 
sponses. Each subject spent three ses- 
sions of approximately 15 minutes with 
the program. A posttest, identical to 
the pretest, was administered on the 
day following completion of the pro- 
gram. 


Results: 

There is no significant difference 
between the groups on either pre- or 
posttest, or in the number of errors 
made on the program. Both groups 
gained approximately 9 points (sta- 
tistically significant) from pretest to 
posttest. These results indicate that 
“learning” did occur; the authors feel, 
however, that the subjects learned to 
understand the test instructions rather 








than the numbe: concept. Study II 
looked into what was learned: 


Study II 


Program: 
Same as Study I. 


Subjects: 
Four children, 5 and 6 years of age, 
students at the Dalton School. 


Procedures: 

This study was run concurrently 
with Study I. Two students were “re- 
inforced,” and two were not. All sub- 
jects completed only the first 87 
frames, which deal with numbers 1, 
2, and 3, and were then given the en- 
tire posttest. 


Results: 

These four children, who took only 
the first third of the program per- 
formed on the pre- and posttest as 
well as the larger group who went 
through the entire program. The au- 
thors rightly make no attempt to gen- 
eralize from these results based on 
minute sample sizes. 

It would seem that either (a) the 
concept of number for the first three 
digits transferred remarkably well to 
the digits four through through nine, 
or (b) the subjects already knew the 
concepts and what they learned was 
how to follow instructions and to 
make comparisons. The authors rec- 
ommend that other researchers investi- 
gating preverbal programs administer 
a preprogram before the experiment 
to teach the children how to follow 
instructions and make comparisons. 


Comment: 

These studies illustrate some of the 
pitfalls which are to be avoided in in- 
vestigating the behavior of young chil- 
dren. They further point to the need 
for clearly defined criteria. 


). 


Keislar, Evan R., & McNeil, John D. 
“Teaching Scientific Theory to First 
Grade Pupils by Auto-Instructional 
Device.” University of California, Los 
Angeles, 1961. 


This study investigated the efficacy 
of an instructional program in teach- 
ing first-grade children to give scien- 
tific explanations of physical phenom- 
ena. 





A 432-frame program designed to 





teach several basic statements of mo- 
lecular theory related to the processes 
of evaporation and condensation was 
presented on a Videosonic Tutor which 
simultaneously projected a slide upon a 
viewing screen and broadcasted a com- 
mentary from a tape recorder. The 
commentary supplied the information 
about each of the slides and asked mul- 
tiple-choice questions. The alternative 
answers were presented as subpictures 
on the slide. After each question the 
pupil pressed a button corresponding to 
the subpicture he chose as his answer. 
The program did not advance until 
the pupil selected the correct answer. 


Subjects: 

Twenty-six Ist-grade pupils, divided 
equally into the experimental group and 
the control group. The two groups 
were matched on the basis of teacher 
ratings, as well as scores on a pretest of 
vocabulary used in the program. 


Procedure: 

The program was broken into 13 
lessons of approximately 36 frames 
each. In the experimental group, each 
child spent approximately 15 to 20 
minutes a day working on the pro- 
gram for a period of 13 days. Chil- 
dren in the control group received no 
special instruction. The day after the 
program was completed, children in 
both groups were given a posttest con- 
sisting of a standardized interview of 
about ten minutes, which, contrary to 
the instructional phase of the study, 
required the learner to say the words 
out loud. 


Results: 

1. All but one of the 13 experi- 
mental children showed higher posttest 
scores than did the contro)s, a difter- 
ence which is statistically significant. 

2. Success on the posttest was sig- 
nificantly related to the number of 
errors made during the instructional 
program (the better the performance 
on the program, the better the per- 
formance on the posttest). 

3. There was some evidence that 
the program was more appropriate for 
children with a high reading readiness 
and effective study habits. 

4. All the children, when asked if 
they would like to continue learning 


new things in this fashion, indicated 
Continued on page 8 








Experienced programers frequently give the novice all 
sorts of untested advice, rules of thumb, recipes, and in- 
tuitions. ‘The advice is based mainly on extrapolations 
from basic learning theory and on each programer’s own 
discovery of what works for him. Rarely do we see a 
clear empirical verification in the classroom of a program- 
ing principle. 

In a field as new as programing, this lack of direct 
evidence is to be expected rather than deplored. The 
“variables” which constitute a program are extremely 
complex, and a workable base-line program, constructed 
according to the extrapolated recipes, must exist before 
research may begin. Variations to be studied can be in- 
troduced systematically and their effects compared with 
the known results of the tested program. Tested pro- 
grams are as yet few in number, but at least one major 
exception does exist. The research described herein is one 
of the rarities and bears on one of the most significant 
rules of thumb that programers know. The investigator 
was Dr. James G. Holland of Harvard University. The 
program was the well-known The Analysis of Behavior, 
co-authored by Dr. Holland and Professor B. F. Skinner. 


In a paper read to the American Psychological Asso- 
ciation convention in 1960, Dr. Holland said: ‘The 
programer’s only contact with the student is in the stu- 
dent’s response, hence the portion of the item left for the 
student to complete is an extremely important choice 
which, faces the programer. All the various aspects of 
programing require that the student give the response for 
the right reason. The response required of the student is 
often chosen only to assure that the critical content or 
the item or perhaps a panel is carefully read or observed. 


For example, success in answering the problem may serve 
only to assure the student’s careful observation of the 
material in front of him at the moment. A term may be 
defined in the first part of an item and an example given 
in the remainder of the item requiring a response which 
can be, given only if the definition has been carefully 
read. .. . The choice of the word to be left blank for 
the student to provide is not arbitrary. It must be one 
which assures that he notice the critical material which 
that item is supposed to teach.” 


A further criterion for choosing the response was that 
“the student should be able to answer the item correct- 
ly”. This part of Dr. Holland’s research will be discussed 
at a later date. 

Items from thirteen lessons of the Holland-Skinner 
program were rewritten to demonstrate the principle 
that students should respond to critical information. No 
changes were made in the information content. The sole 
change lay in the response the student was required to 
give, 


For example, in Lesson 9, item 2 introduces a key 


term. The information part of the item reads: A technical 
term for “reward” is reinforcement. The critical informa- 
tion is obviously the new term. 


According to the “recipes” offered by all programers, 
the student learns what he is led to do, and, if he is to 
learn ‘reinforcement,’ he should make some response to 
this term. Following this procedure, the original item in 
the true version of the program reads: A technical term 
for “reward” is reinforcement. To “reward” an organism 
with food is to it with food. As given, the item 
requires the student to attend to the new term. 


The faulty version, as deliberately butchered by Dr. 
Holland reads: A technical term for “reward” is rein- 
forcement. To “reward” an organism with food is to rein- 
force it with 

In this version, the student is not required to attend 
to the new term. The programer has no “written guar- 
antee’ that the student has read the word at all. A 
longer illustration of this type of noncritical response is 
provided in parallel with the true version. 


True version: Lesson 11 
1. A stimulus which follows a response is called a 
if the rate at which similar responses are 
emitted is observed to increase. (Answer: reinforcer) 
2. A hungry pigeon pecks a key and is immediately 
given food. The of pecking will increase 
since presenting food a reinforcement. (An- 
swer: rate, is) 


3. If, instead of presenting food after the pigeon 
pecks the key, a loud noise is turned on, the rate 
of pecking will not increase. Presenting a loud 
noise_______a reinforcement. (Answer: is not) 


4. When pecking a key turns off a very loud noise 
for a few moments, the frequency of pecking in the 
presence of the noise is observed to increase. 
Ending the loud noise a reinforcement. 
(Answer: is) 

5. Reinforcement which consists of presenting stimuli 
(e.g., food) is called positive reinforcement. In con- 
trast, reinforcement which consists of terminating 
stimuli (e.g., painful stimuli) is called rein- 
forcement. (Answer: negative) 


6. Turning off a television commercial is reinforced by 
the termination of the reinforcer; turning on 
a very funny program is reinforced by the presenta- 
tion of a reinforcer. (Answer: negative, pos- 
itive) 











7. A stimulus is called a negative reinforcer if it rein- 


forces the bebavior which it. (Answer: 


terminates) 

8. Elimination of a television commercial may be a 
negative reinforcement. The television commercial 
is a - (Answer: negative reinforcer) 


9. If a funny program is a positive reinforcer, present- 
ing the program is a (Answer: pos- 
tive reinforcement) 


10. The rate of an operant can be increased by pre- 
senting a or by ending @ 
. (Answer: positive reinforcer, negative re- 

inforcer) 
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The critical content of the lesson presented here is 
obviously the common properties of and the differences 
between the operations called “positive reinforcement” 
and “negative reinforcement.” (The reader should not 
be disturbed if he finds the good version difficult; it is 
well along in the program and assumes a good deal of 
previous learning.) The true version of the program re- 
quires the student to make responses relevant to these 
concepts. 


The faulty version contains some clever parodies of 
items, such as requiring a sophisticated student to respond 
“pigeon” and “key” (no. 3), or “commercial” and “tele- 
vision” (no. 8). 

Some of the items are less obviously noncritical re- 
sponses. In items 1, 5, and 7, the responses are technical 
terms, but these responses can be given on the basis of 
previous learning by a student who does not understand 
the distinction between negative and positive reinforce- 
ment. Most of the items can be answered by a good 
reader on the basis of their grammar alone, with no 
understanding required. 


Faulty version: Lesson 11 

i. A stimulus which follows a response is called a 
reinforcer if the rate at which similar are 
ormsted is observed to increase. (Answeff: responses) 


2. A hungry pigeon pecks a key and 4¥ Immediately 
given food. The rate of pecking w rease since 


preseming Li gnendicecge: aa reinforcement. (Answer: 





1 
3. instal: of presenting food: after the 
suibs the _ @ loud nojs@.is turned on, the 





rate of packing will not increase. Presenting a | a loud 
noise is nota einforcement, (Answer: pigeon, key) 


4. When pecking @ key tums offs very loud noise for 
& few moments, frequency of pecking in the pres- 
ence of the nols@ is observed to increase. Ending 
the loud —__ b reinforcement. (Answer: noise) 


5. Reinforcement whith €Onsists of presenting stimuli 
sitive reintorcenant In con- 


(e.g., food) is called pes 
trast, reinforcement which consists of terminating 
(e.g., painful stimuli) is called negative re- 


inforcement. (Answer: stimuli) 


6. Turning off a tetevision commercial is reinforced by 
the termination of the negative reinforcer; turning 
on a very funny program ig.__ed by the pre- 

sentation of | positive relegeaber. (Answer: rein- 


forced) mao Be 


7. A stimulds. is called a negative. yeinforcer if it 
—___/ behavior which terminate ie ee: re- 


ae 






















reinforcer. (Answer: conamatel al, te 


ae a positive reinforcer, p 
e program is a positive reinforcer nent. (An- 
swéf: program) AR: 
10. The rate of an operant can be increased by piltond: 


ing a positive reinforcer or by ending a negative 
(Answer: reinforcer) 





It should be noted that the error rate found by Dr. 
Holland on the faulty version was less than half that 
found on the true version. An item which can be an- 
swered on the strength of its grammatical structure is 
not as difficult as an item which requires understanding 
of the material. A reduction in error rate is desirable if it 
is not accompanied by a reduction in acquisition. In terms 
of the final performance of the students on a test of their 
knowledge, the results strongly favor those students who 
were required to interact with the critical aspects of ma- 
terial. 

In the last issue of the Bulletin, this column discussed 
the effects of sequencing on the error rate and on final 
achievement. Evidence reported there suggested that low 
error rates are not always reflected in superior perform- 
ance on a final test. Dr. Holland’s data support this 
conclusion. Again, overdependence on grammar and un- 
derdependence on understanding lead to poor final 
performance. Correct responses are valuable only when 
the student has been led to do what the programer wishes 
to teach him. In Dr. Holland’s words: “Items must be 
so written that the ability to correctly answer the item 
depends on noticing the critical information in the item.” 
Programers should get from the student a “written 
guarantee” that he understands. Evidence that he reads 
English is not enough. 

SuUsAN MEYER MARKLE 
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EXPERIMENTATION WITH 
PROGRAMED INSTRUCTION 


(Continued from page 3) 


problems exist, of course, in a// instructional situations; 
programing makes them inescapable. 


Careful observation of a class using a programed unit 
provides an opitimal way of discovering what programed 
instruction is. Many teachers will also find the record of 
each student’s results an excellent basis for diagnosing 
his individual learning difficulties. 

Consumers must be able to assess the quality of pro- 
grams for themselves. If the publisher has presented data 
about the efficacy of the program, the consumer can com- 
pare his results with those of the publisher’s. Some 
school systems are already involved in field testing pro- 
grams in cooperation with programers and publishers who 
are developing and refining programs. 

To aid school personnel interested in systematic ex- 
ploration of some of the problems in the field the Cen- 
ter will hold weekly seminars to discuss experimentation 
with and implementation of programs. (See announce- 
ment, page 8.) 

The experimental approaches discussed in this article 
explore programed instruction under conditions that seek 
to minimize the influence of the teacher, thus allowing 
the program to demonstrate what it can do on its own. 

In the next issue of the Bulletin we will discuss im- 
plementation studies which try to answer the question, 
“How can a teacher use programs most effectively ? 


LAssAR GOTKIN 








RING OUT (Continued from page 1) 
able” student and parent reaction to 
the materials. ‘When we searched for 
negative reactions we could find very 
few; the students have, however, sug- 
gested greater use of demonstrations 
and experiments to parallel the texts.” 

To all those who have visited the 
Center during the past year, assisted 
with research and field trials, partici- 
pated in our workshops, sat through 
our speeches, read our written sym- 
bols, provided “‘stimuli” letters, the 
following would like to say, Thank 
you, and to send you best wishes for 
the holiday season. 
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Bernard Basescu 
David C. Belash 
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Ruth Emerson 

Carl Epstein* 
Eleanor Feld* 
Renee G. Ford 
Heyta Franke 
Robert Horn 
Jerome D. Kaplan 
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Rita A. Ragger* 
Syrell Rogovin 
Ze’ev Rosenfeld 
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Audrey Sterenfeld 
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Seth Wohl 
Karl E. Zimmer 
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Information Division 

Marion V. Bell 


Carol Christman 
Lincoln Hanson 
Sara N. Thompson 
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© assist persons interested in ex- 
ing with programed instruc- 
“tion, the Center’s research staff will 
hold an informal seminar every Thurs- 
day. afternoon from 3 to 5 o'clock. 
‘Schoo! administrators, teachers, grad- 


wate students, and others. seeking guid- 







re in the experimental investigation 
ree instruction are invited to 


at the seminar “it be 


‘sons are, therefore, advised to contact 
Mrs. Sally Thompson at the Center 

1, 365. West 
ork City 24, 


RECENT RESEARCH 


(Continued from page 5) 


that they would. Their classroom 
teacher reported that they looked for- 
ward to the program each day. 


Comment: 

The results of this study are quite 
encouraging. It is interesting to note 
that in this study a multiple-choice 
technique was used effectively. The 
authors’ method of testing the chil- 
dren may be a significant contribution 
to this field in which the testing of 
very young children creates consider- 
able problems. This supports the find- 
ings reviewed in this bulletin, as well 
as findings in a number of other stud- 
ies involving subjects who could read 
and write. 


RESPONSES 
(Continued from page 2) 





Even though a teacher understands 
the material and uses good methods in 
presentation, a teacher seldom has both 
time and the extensive knowledge to 
produce an effective program. The 
suggestions of an experienced teacher 
are valuable and should be incorporat- 
ed in revisions of the program. 


Loetta W. Horton 


I do feel that the basic work in pre- 
paring the programs, done by subject 
matter specialists, followed by teacher’s 
comments made after using the ma- 
terial, is much better than having 
high school teachers prepare their own 
programs. 


A subject matter specialist knows 
better than most high school teachers 
exactly what the pupil has to contend 
with beyond the high school in that 
particular subject and can see that all 
necessary background material has been 
included. 

Also, the subject matter specialists 
are closer to the fields where that par- 
ticular subject will be put into use, 
while the heavy schedules most high 
school teachers carry does not allow 
much time for research. 

The specialist, I feel, by making the 
tests, stresses important facts at the 
most effective points, and by stating 
where each particular test should be 
given, helps the teacher. 

Florence Tucker 


More Recent Research .... 


The road to Serendip is paved with many a 
slip. Noel Korn of Valley College in California 
reminds us that Horace, not Hugh Walpole, 
deserves credit for sending the princes to 
Serendip as cited in the October issue. 
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